Theoretical Foundations of Linear and Order Statistics Combiners for Neural Pattern Classifiers

Several researchers have experimentally shown that substantial improvements can be obtained in diicult pattern recognition problems by combining or integrating the outputs of multiple classiiers. This paper provides an analytical framework to quantify the improvements in classiication results due to combining. The results apply to both linear combiners and the order statistics combiners introduced in this paper. We show that combining networks in output space reduces the variance of the actual decision region boundaries around the optimum boundary. For linear combiners, we show that in the absence of classiier bias, the added classiication error is proportional to the boundary variance. For non-linear combiners, we show analytically that the selection of the median, the maximum and in general the ith order statistic improves classiier performance. The analysis presented here facilitates the understanding of the relationships among error rates, classiier boundary distributions, and combining in output space. Experimental results on several public domain data sets are provided to illustrate the beneets of combining.